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Why  This  Workshop? 

Introduction  to  CMMI  Process  Performance  Models  and  Baselines 
Contrasting  Large  vs  Small  Organizational  Settings  (group  exercises) 

1.  Project  Lifecycle  Needs 

2.  Performance  Outcomes  (“y’s”) 

3.  "x"  Factors  (controllable  and  un-controllable) 

4.  Usage  of  Models 

5.  Analytical  Methods 

6.  Training  and  Deployment 

7.  Sponsorship  and  Participation 

Next  Steps 


Interpreting  CMMI  High  Maturity  for  Small 
Organizations 
Robert  W.  Stoddard  II 
September,  2008 

©  2008  Carnegie  Mellon  University 


Software  Engineering  Institute  Carnegie  Mellon 


2 


Why  This  Workshop? 


Congreso  Internacional  en  Ingenieria  de  Software  y  sus  Aplicaciones 
(International  Congress  of  Software  Engineering  and  its  Applications) 


CMMI  Process  Performance  Models  and  Baselines  are  not  clearly 
understood 

-  historical  misconceptions  resulting  in  lackluster  results 

-  opportunity  to  leverage  proven  Six  Sigma  toolkit 

Confusion  exists  regarding  the  applicability  of  CMMI  Process 
Performance  Models  and  Baselines  to  small  organizational  settings 

Small  settings  in  this  workshop  refers  to  projects  of  3-9  months 
duration  with  3-10  staff 

Performance  results  must  be  elevated  above  compliance  to  a  given 
model 
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Congreso  Internacional  en  Ingenieria  de  Software  y  sus  Aplicaciones 
(International  Congress  of  Software  Engineering  and  its  Applications) 


INTRODUCTION  TO  CMMI 
PROCESS  PERFORMANCE 
MODELS  AND  BASELINES 
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OPP  SP1.1 


Congreso  Internacional  en  Ingenieria  de  Software  y  sus  Aplicaciones 
(International  Congress  of  Software  Engineering  and  its  Applications) 


Select  Processes 


Select  the  processes  or  subprocesses  in  the  organization’s  set  of 
standard  processes  that  are  to  be  included  in  the  organization’s 
process-performance  analyses. 


Select  processes/subprocesses  that  will  help  us  understand  our 
ability  to  meet  the  objectives  of  the  organization  and  projects,  and  the 
need  to  understand  quality  and  process  performance.  These 
subprocesses  will  typically  be  the  major  contributors  and/or  their 
measures  will  be  the  leading  indicators. _ 


Excerpted  from  Tutorial:  “If  You’re  Living  the  “High  Life”,  You’re  Living  the  Informative  Material”  presented  at  the  SEPG 
North  America,  by  Rusty  Young,  Mike  Konrad  and  Bob  Stoddard,  March,  2008 
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OPP  SP  1.2 
Measures 


Congreso  Internacional  en  Ingenieria  de  Software  y  sus  Aplicaciones 
(International  Congress  of  Software  Engineering  and  its  Applications) 


Establish  Process-Performance 


Establish  and  maintain  definitions  of  the  measures  that  are  to  be 
included  in  the  organization’s  process-performance  analyses. 


Select  measures,  analyses,  and  procedures  that  provide  insight  into 
the  organization’s  ability  to  meet  its  objectives  and  into  the 
organization’s  quality  and  process  performance.  Create/update  clear 
unambiguous  operational  definitions  for  the  selected  measures. 
Revise  and  update  the  set  of  measures,  analyses,  and  procedures  as 
warranted.  In  usage,  be  sensitive  to  measurement  error.  The  set  of 
measures  may  provide  coverage  of  the  entire  lifecycle  and  be 
controllable. 


Excerpted  from  Tutorial:  “If  You’re  Living  the  “High  Life”,  You’re  Living  the  Informative  Material”  presented  at  the  SEPG 
North  America,  by  Rusty  Young,  Mike  Konrad  and  Bob  Stoddard,  March,  2008 
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Congreso  Internacional  en  Ingenieria  de  Software  y  sus  Aplicaciones 
(International  Congress  of  Software  Engineering  and  its  Applications) 


OPP  SP  1.3  Establish  Quality  and  Process- 
Performance  Objectives 


Establish  and  maintain  quantitative  objectives  for  quality  and  process 
performance  for  the  organization. 


These  objectives  will  be  derived  from  the  organization’s  business 
objectives  and  will  typically  be  specific  to  the  organization,  group,  or 
function.  These  objectives  will  take  into  account  what  is  realistically 
achievable  based  upon  a  quantitative  understanding  (knowledge  of 
variation)  of  the  organization’s  historic  quality  and  process 
performance.  Typically  they  will  be  SMART  and  revised  as  needed. 


Excerpted  from  Tutorial:  “If  You’re  Living  the  “High  Life”,  You’re  Living  the  Informative  Material”  presented  at  the  SEPG 
North  America,  by  Rusty  Young,  Mike  Konrad  and  Bob  Stoddard,  March,  2008 
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OPP  SP  1.4 
Baselines 


Congreso  Internacional  en  Ingenieria  de  Software  y  sus  Aplicaciones 
(International  Congress  of  Software  Engineering  and  its  Applications) 


Process-Performance 


Establish  and  maintain  the  organization's  process-performance 
baselines. 


Baselines  will  be  established  by  analyzing  the  distribution  of  the  data 
to  establish  the  central  tendency  and  dispersion  that  characterize  the 
expected  performance  and  variation  for  the  selected 
process/subprocess.  These  baselines  may  be  established  for  single 
processes,  for  a  sequence  of  processes,  etc.  When  baselines  are 
created  based  on  data  from  unstable  processes,  it  should  be  clearly 
documented  so  the  consumers  of  the  data  will  have  insight  into  the 
risk  of  using  the  baseline.  Tailoring  may  affect  comparability  between 
baselines. 


Excerpted  from  Tutorial:  “If  You’re  Living  the  “High  Life”,  You’re  Living  the  Informative  Material”  presented  at  the  SEPG 
North  America,  by  Rusty  Young,  Mike  Konrad  and  Bob  Stoddard,  March,  2008 


=  Software  Engineering  Institute  Carnegie  Mellon 
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OPP  SP  1.5 
Models 
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Process-Performance 


Establish  and  maintain  the  process-performance  models  for  the 
organization’s  set  of  standard  processes. 


Rather  than  just  a  point  estimate,  PPMs  will  address  variation  in  the 
prediction.  PPMs  will  model  the  interrelationships  between 
subprocesses  including  controllable/uncontrollable  factors.  They 
enable  predicting  the  effects  on  downstream  processes  based  on 
current  results.  They  enable  modeling  of  a  PDP  to  predict  if  the 
project  can  meet  its  objectives  and  evaluate  various  alternative  PDP 
compositions.  They  can  predict  the  effects  of  corrective  actions  and 
process  changes.  They  can  also  be  used  to  evaluate  the  effects  of 
new  processes  and  technologies/innovations  in  the  OSSP. 


Excerpted  from  Tutorial:  “If  You’re  Living  the  “High  Life”,  You’re  Living  the  Informative  Material”  presented  at  the  SEPG 
North  America,  by  Rusty  Young,  Mike  Konrad  and  Bob  Stoddard,  March,  2008 
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QPM  SP  1.1 


Congreso  Internacional  en  Ingenieria  de  Software  y  sus  Aplicaciones 
(International  Congress  of  Software  Engineering  and  its  Applications) 


the  Project’s  Objectives 


Establish  and  maintain  the  project’s  quality  and  process-performance 
objectives. 


These  objectives  will  be  based  on  the  organization’s  quality  and 
process  performance  objectives  and  any  additional  customer  and 
relevant  stakeholder  needs  and  objectives.  These  objectives  will  be 
realistic  (based  upon  analysis  of  historical  quality  and  process 
performance)  and  will  cover  interim,  supplier,  and  end-state 
objectives.  Conflicts  between  objectives  (i.e.,  trade-offs  between 
cost,  quality,  and  time-to-market)  will  be  resolved  with  relevant 
stakeholders.  Typically  they  will  be  SMART,  traceable  to  their  source, 
and  revised  as  needed. 


Excerpted  from  Tutorial:  “If  You’re  Living  the  “High  Life”,  You’re  Living  the  Informative  Material”  presented  at  the  SEPG 
North  America,  by  Rusty  Young,  Mike  Konrad  and  Bob  Stoddard,  March,  2008 


=  Software  Engineering  Institute  Carnegie  Mellon 
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QPM  SP  1.2  Compose 


Congreso  Internacional  en  Ingenieria  de  Software  y  sus  Aplicaciones 
(International  Congress  of  Software  Engineering  and  its  Applications) 


the  Defined  Process 


Select  the  subprocesses  that  compose  the  project’s  defined  process 
based  on  historical  stability  and  capability  data. 


The  PDP  is  composed  by: 

•  selecting  subprocesses 

•  adjusting/trading-off  the  level  and  depth  of  intensity  of 
application  of  the  subprocess(es)  and/or  resources 

to  best  meet  the  quality  and  process  performance  objectives.  This 
can  be  accomplished  by  modeling/simulating  the  candidate  PDP(s)  to 
predict  if  they  will  achieve  the  objectives,  and  the  confidence  level  of 
(or  risk  of  not)  achieving  the  objective. 


Excerpted  from  Tutorial:  “If  You’re  Living  the  “High  Life”,  You’re  Living  the  Informative  Material”  presented  at  the  SEPG 
North  America,  by  Rusty  Young,  Mike  Konrad  and  Bob  Stoddard,  March,  2008 


=  Software  Engineering  Institute  Carnegie  Mellon 
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Congreso  Internacional  en  Ingenieria  de  Software  y  sus  Aplicaciones 
(International  Congress  of  Software  Engineering  and  its  Applications) 


QPM  SP  1.3  Select  the  Subprocesses  that  Will 
Be  Statistically  Managed 


Select  the  subprocesses  of  the  project's  defined  process  that  will  be 
statistically  managed. 


Subprocesses  that  are  the  major  contributors  to  or  predictors  of  the 
accomplishment  of  the  project’s  interim  or  end-state  objectives  will  be 
selected.  Additionally,  these  need  to  be  suitable  for  statistical 
management.  Statistically  managing  the  selected  subprocesses 
provides  valuable  insight  into  performance  by  helping  the  project 
identify  when  corrective  action  is  needed  to  achieve  its  objectives. 
Select  the  attributes  that  will  measured  and  controlled. 


Excerpted  from  Tutorial:  “If  You’re  Living  the  “High  Life”,  You’re  Living  the  Informative  Material”  presented  at  the  SEPG 
North  America,  by  Rusty  Young,  Mike  Konrad  and  Bob  Stoddard,  March,  2008 
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Congreso  Internacional  en  Ingenieria  de  Software  y  sus  Aplicaciones 
(International  Congress  of  Software  Engineering  and  its  Applications) 


QPM  SP  1.4  Manage  Project  Performance 


Monitor  the  project  to  determine  whether  the  project’s  objectives  for 
quality  and  process  performance  will  be  satisfied,  and  identify 
corrective  action  as  appropriate. 


Monitor  the  project 

•  Manage  stability  and  capability  of  selected  subprocesses. 

•  Track  quality  and  process  performance  data  including  suppliers’ 

•  Update/calibrate  PPMs  and  predictions  based  on  results  to  date. 

•  Identify  deficiencies/risks  to  achieving  objectives  (e.g.,  where 

current  performance  is  outside  tolerance  intervals,  or 
prediction/confidence  intervals  are  not  contained  within 
specification  limits). _ 


Excerpted  from  Tutorial:  “If  You’re  Living  the  “High  Life”,  You’re  Living  the  Informative  Material”  presented  at  the  SEPG 
North  America,  by  Rusty  Young,  Mike  Konrad  and  Bob  Stoddard,  March,  2008 
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Congreso  Internacional  en  Ingenieria  de  Software  y  sus  Aplicaciones 
(International  Congress  of  Software  Engineering  and  its  Applications) 


QPM  SP  2.1  Select  Measures  and  Analytic 
Techniques 


Select  the  measures  and  analytic  techniques  to  be  used  in  statistically 
managing  the  selected  subprocesses. 


Identify  the  measures  that  will  provide  insight  into  the  performance  of 
the  subprocesses  selected  for  statistical  management  and  the 
statistical  techniques  that  will  be  used  for  analysis.  These  measures 
can  be  for  both  controllable  and  uncontrollable  factors.  Operational 
definitions  will  be  created/updated  for  these  measures.  Where 
appropriate  (i.e.,  they  are  critical  to  meeting  downstream  objectives), 
spec  limits  will  be  established  for  the  measures. 


Excerpted  from  Tutorial:  “If  You’re  Living  the  “High  Life”,  You’re  Living  the  Informative  Material”  presented  at  the  SEPG 
North  America,  by  Rusty  Young,  Mike  Konrad  and  Bob  Stoddard,  March,  2008 
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Congreso  Internacional  en  Ingenieria  de  Software  y  sus  Aplicaciones 
(International  Congress  of  Software  Engineering  and  its  Applications) 


QPM  SP  2.2  Apply  Statistical  Methods  to 
Understand  Variation 


Establish  and  maintain  an  understanding  of  the  variation  of  the 
selected  subprocesses  using  the  selected  measures  and  analytic 
techniques. 


Selected  measures  for  the  subprocesses  will  be  statistically 
controlled  to  identify,  remove,  and  prevent  reoccurrence  of  special 
causes  of  variation,  or  in  other  words,  stabilize  the  process.  When 
control  limits  are  too  wide,  sources  of  variation  are  easily  masked  and 
further  investigation  is  warranted. _ 


Excerpted  from  Tutorial:  “If  You’re  Living  the  “High  Life”,  You’re  Living  the  Informative  Material”  presented  at  the  SEPG 
North  America,  by  Rusty  Young,  Mike  Konrad  and  Bob  Stoddard,  March,  2008 


=  Software  Engineering  Institute  Carnegie  Mellon 
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Congreso  Internacional  en  Ingenieria  de  Software  y  sus  Aplicaciones 
(International  Congress  of  Software  Engineering  and  its  Applications) 


QPM  SP  2.3  Monitor  Performance  of  the 
Selected  Subprocesses 


Monitor  the  performance  of  the  selected  subprocesses  to  determine 
their  capability  to  satisfy  their  quality  and  process-performance 
objectives,  and  identify  corrective  action  as  necessary. 


Fora  stable  subprocess,  determine  if  the  control  limits  (natural 
bounds)  are  within  the  specification  limits  which  indicates  a  capable 
subprocess.  If  it  is  not,  document  corrective  actions  that  address  the 
capability  deficiencies. 


Excerpted  from  Tutorial:  “If  You’re  Living  the  “High  Life”,  You’re  Living  the  Informative  Material”  presented  at  the  SEPG 
North  America,  by  Rusty  Young,  Mike  Konrad  and  Bob  Stoddard,  March,  2008 
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Congreso  Internacional  en  Ingenieria  de  Software  y  sus  Aplicaciones 
(International  Congress  of  Software  Engineering  and  its  Applications) 


When  and  Why  Do  We  Need  Process 
Performance  Models  at  the  Project  Level? 
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Process  Performance  Models  View  Processes 
Holistically 

Processes  may  be  thought  of  holistically  as  a  system  that  includes  the 
people,  materials,  energy,  equipment,  and  procedures  necessary  to 
produce  a  product  or  service. 


Requirements 
&  Ideas 


People  Material  Energy  Equipment  Procedures 


Q  Q  Q  Q  Q 


Work  Activities  nn 

K 

> 

Time 

1/ 

v  Products  & 
^  Services 


=  Software  Engineering  Institute  Carnegie  Mellon 


Interpreting  CMMI  High  Maturity  for  Small 
Organizations 
Robert  W.  Stoddard  II 
September,  2008 

©2008  Carnegie  Mellon  University 


18 


Congreso  Internacional  en  Ingenieria  de  Software  y  sus  Aplicaciones 
(International  Congress  of  Software  Engineering  and  its  Applications) 


Healthy  Ingredients  of  CMMI  Process 
Performance  Models 


i.  Statistical,  probabilistic  or  simulation  in  nature 


2.  Predict  interim  and/or  final  project  outcomes 

3.  Use  controllable  factors  tied  to  sub-processes  to  conduct  the 
prediction 

4.  Model  the  variation  of  factors  and  understand  the  predicted  range  or 
variation  of  the  outcomes 

5.  Enable  “what-if”  analysis  for  project  planning,  dynamic  re-planning 
and  problem  resolution  during  project  execution 


6. 

7. 


Connect  “upstream”  activity  with  “downstream”  activity 

Enable  projects  to  achieve  mid-course  corrections  to  ensure  project 
success 
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All  Models  (Qualitative  and  Quantitative) 

Quantitative  Models  (Deterministic,  Statistical,  Probabilistic) 


Statistical  or  Probabilistic  Models 


Interim  outcomes  predicted 
Controllable  x  factors  involved 


Process  Performance 
Model  - 

With  controllable  x 
factors  tied  to 
Processes  and/or  Sub¬ 
processes 


Only  phases 
or  lifecycles 
are  modeled 


Only 

uncontrollable 
factors  are 
modeled 


Only  final 
outcomes 
are 

modeled 


No 

uncertainty 
or  variation 
modeled 


Anecdotal 

Biased 

samples 
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CONTRASTING  LARGE  VS 
SMALL  ORGANIZATIONAL 
SETTINGS: 

PROJECT  LIFECYCLE  NEEDS 
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Project  Lifecycle  Needs 


Large  Settings 

•  Distinct  phases  and 
activities  performed  in 
specified  serial  fashion 

•  Different  people  or  teams 
involved  in  the  different 
phases  and  activities 

•  Risks  during  internal  hand- 
offs  quite  great 

•  Communication  and 
expectations  not  matched 


Small  Settings 

•  Fluid  phases  and 
processes  running  together 

•  Same  people  perform 
many  if  not  most  of  the 
activities 

•  Risks  between  external 
entities  are  greatest 

•  Lack  of  cross  training  is 
high  risk;  depend  on 
specific  individuals 
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Group  Exercise  #1 
(10  minutes) 


Congreso  Internacional  en  Ingenieria  de  Software  y  sus  Aplicaciones 
(International  Congress  of  Software  Engineering  and  its  Applications) 


Within  your  group,  share  ideas  on  what  the  most  important  lifecycle 
needs  and  risks  are  in  your  small  organizational  settings 


Record  your  group  ideas  on  your  group  flip  pad 


Prepare  to  share  3-5  ideas  with  the  audience  at  large 
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Congreso  Internacional  en  Ingenieria  de  Software  y  sus  Aplicaciones 
(International  Congress  of  Software  Engineering  and  its  Applications) 


CONTRASTING  LARGE  VS 
SMALL  ORGANIZATIONAL 
SETTINGS: 

PERFORMANCE  OUTCOMES 
ITS) 
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Performance 


Large  Settings 

•  Final  project  quality, 
schedule  and  cycle  time 
measures 

•  Interim  outcomes  tied  to 
key  phase  and  activity 
hand-offs 

•  Communication  across 
groups  and  geographic 
locations 


Software  Engineering  Institute 


(Vs) 


Small  Settings 

•  Customer  Satisfaction 

•  Customer  Relationship 

•  Req'ts  Completeness  and 
Understanding 

•  Relationship  with  suppliers 
or  other  subcontractors 

•  Availability  of  key  staff 

•  Staff  versatility,  training 

•  Staff  Productivity,  Morale 
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Performance  Outcomes  (“y’s) 


Large  Settings 

•  Final  project  quality, 
schedule  and  cycle  time 

Think  of  the  outcomes  that 
would  benefit  a  small  project  if 
they  had  the  ability  to  predict 
and  re-predict  during  their 
short  lifecycle  to  maximize 
,  success 

cr "T - -  cj - cj  ■  —  r  ■  " 

locations 


Small  Settings 

Customer  Satisfaction 

Customer  Relationship 

Req'ts  Completeness  and 
Understanding 

Relationship  with  suppliers 
or  other  subcontractors 

Availability  of  key  staff 

Staff  versatility,  training 


•  Staff  Productivity,  Morale 
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Group  Exercise  #2 
(10  minutes) 


Congreso  Internacional  en  Ingenieria  de  Software  y  sus  Aplicaciones 
(International  Congress  of  Software  Engineering  and  its  Applications) 


Within  your  group,  discuss  the  types  of  performance  outcomes  that 
your  projects,  within  small  settings,  are  most  concerned  with. 


Document  the  ideas  on  your  group  flip  pad 


Be  prepared  to  share  some  of  these  with  the  audience  at  large 
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Congreso  Internacional  en  Ingenieria  de  Software  y  sus  Aplicaciones 
(International  Congress  of  Software  Engineering  and  its  Applications) 


CONTRASTING  LARGE  VS 
SMALL  ORGANIZATIONAL 
SETTINGS: 

“X”  FACTORS  (CONTROLLABLE 
AND  UNCONTROLLABLE) 
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Congreso  Internacional  en  Ingenieria  de  Software  y  sus  Aplicaciones 
(International  Congress  of  Software  Engineering  and  its  Applications) 


Data  Types  Determine  Which  Techniques  To  Use 


Categorical  data  where  the  order  of  the 
Nominal  categories  is  arbitrary 


Attribute 

(aka  categorized  or 
discrete  data) 

Nominal  data  with  an  ordering;  may 
Ordinal  have  unequal  intervals 


Examples 
Defect  types 
Labor  types 
Languages 

Examples 
Severity  levels 
Survey  choices  1-5 
Experience  categories 


Continuous 
(aka  variables 
data) 


Interval  Continuous  data  with  equal  intervals;  may 
have  decimal  values 


Ratio 


Examples 
Defect  densities 
Labor  rates 
Productivity 
Variance  %’s 
Code  size  SLOC 


Interpreting  CMMI  High  Maturity  for  Small 
Organizations 
Robert  W.  Stoddard  II 
September,  2008 

©  2008  Carnegie  Mellon  University 


Software  Engineering  Institute  Carnegie  Mellon 


29 


ANOVA  &  Dummy  Variable  Regression  Models 


Using  these  controllable  factors... 

To  predict  this  outcome! 

Type  of  Reviews  Conducted;  Type  of  Design 
Method;  Language  Chosen;  Types  of  Testing 

Delivered  Defect  Density 

High-Medium-Low  Domain  Experience; 

Architecture  Layer;  Feature;  Team;  Lifecycle 
model;  Primary  communication  method 

Productivity 

Estimation  method  employed;  Estimator;  Type  of 
Project;  High-Medium-Low  Staff  Turnover;  High- 
Medium-Low  Complexity;  Customer;  Product 

Cost  and  Schedule 

Variance 

Team;  Product;  High-Medium-Low  Maturity  of 
Platform;  Maturity  or  Capability  Level  of  Process; 
Decision-making  level  in  organization;  Release 

Cycle  Time  or 
Time-to-Market 

Iterations  on  Req’ts;  Yes/No  Prototype;  Method  of 
Req’ts  Elicitation;  Yes/No  Beta  Test;  Yes/No  On- 
Time;  High-Medium-Low  Customer  Relationship 

Customer  Satisfaction  (as 
a  percentile  result) 

Simple  and  Multiple  Regression 


Using  these  controllable  factors... 

To  predict  this 
outcome! 

Req’ts  Volatility;  Design  and  Code  Complexity; 
Test  Coverage;  Escaped  Defect  Rates 

Delivered  Defect  Density 

Staff  Turnover  %;  Years  of  Domain  Experience; 
Employee  Morale  Survey  %;  Volume  of 
Interruptions  or  Task  Switching 

Productivity 

Availability  of  Test  Equipment  %;  Req’ts 

Volatility;  Complexity;  Staff  Turnover  Rates 

Cost  and  Schedule 
Variance 

Individual  task  durations  in  hrs;  Staff  availability 
%;  Percentage  of  specs  undefined;  Defect 
arrival  rates  during  inspections  or  testing 

Cycle  Time  or 

Time-to- Market 

Resolution  time  of  customer  inquiries; 

Resolution  time  of  customer  fixes;  Percent  of 
features  delivered  on-time;  Face  time  per  week 

Customer  Satisfaction 
(as  a  percentile  result) 
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Chi-Square  &  Logistic  Regression 


Using  these  controllable  factors... 

To  predict  this  outcome! 

Programming  Language;  High-Medium-Low 
Schedule  compression;  Req’ts  method;  Design 
method;  Coding  method;  Peer  Review  method 

Types  of  Defects 

Predicted  Types  of  Defects;  High-Medium-Low 
Schedule  compression;  Types  of  Features 
Implemented;  Parts  of  Architecture  Modified 

Types  of  Testing  Most 
Needed 

Architecture  Layers  or  components  to  be 
modified;  Type  of  Product;  Development 
Environment  chosen;  Types  of  Features 

Types  of  Skills  Needed 

Types  of  Customer  engagements;  Type  of 
Customer;  Product  involved;  Culture;  Region 

Results  of  Multiple  Choice 
Customer  Surveys 

Product;  Lifecycle  Model  Chosen;  High-Medium- 
Low  Schedule  compression;  Previous  High  Risk 
Categories 

Risk  Categories  of  Highest 
Concern 

Logistic  Regression 


Using  these  controllable  factors... 

To  predict  this 
outcome! 

Inspection  Preparation  Rates;  Inspection  Review 
Rates;  Test  Case  Coverage  %;  Staff  Turnover 

Rates;  Previous  Escape  Defect  Rates 

Types  of  Defects 

Escape  Defect  Rates;  Predicted  Defect  Density 
entering  test;  Available  Test  Staff  Hours;  Test 
Equipment  or  Test  Software  Availability 

Types  of  Testing  Most 
Needed 

Defect  Rates  in  the  Field;  Defect  rates  in  previous 
release  or  product;  Turnover  Rates;  Complexity  of 
Issues  Expected  or  Actual 

Types  of  Skills  Needed 

Time  (in  Hours)  spent  with  Customers;  Defect 
rates  of  products  or  releases;  Response  times 

Results  of  Multiple  Choice 
Customer  Surveys 

Defect  densities  during  inspections  and  test;  Time 
to  execute  tasks  normalized  to  work  product  size 

Risk  Categories  of 

Highest  Concern 

“x”  Factors 


Large  Settings 

•  Reqts  Volatility 

•  Architecture  and  Design 
complexity 

•  Code  complexity 

•  Test  Coverage 

•  Test  Execution 

•  Avg  experience  level  of 
team 


Small  Settings 

•  People  attributes  such  as: 

•  Personal  productivity 

•  Individual  interruptions 

•  Teaming  Attributes 

•  Conflict  resolution 

•  Domain  experience  of  key 
staff 

•  Knowledge  sharing  methods 

•  Daily  communications 


•  Modern  development  tools 
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“x”  Factors 


Large  Settings  Small  Settings 

•  Reqts  Volatility  •  People  attributes  such  as: 


Architecture  and  Design 

Think  of  the  “x”  factors  related 
to  individual  and  small  team 
activities  that  drive 
performance  outcomes 

Test  Execution 

Avg  experience  level  of 
team 


Modern  development  tools 


•  Personal  productivity 

•  Individual  interruptions 

•  Teaming  Attributes 

•  Conflict  resolution 

•  Domain  experience  of  key 
staff 

•  Knowledge  sharing  methods 

•  Daily  communications 
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Group  Exercise  #3 
(10  minutes) 


Congreso  Internacional  en  Ingenieria  de  Software  y  sus  Aplicaciones 
(International  Congress  of  Software  Engineering  and  its  Applications) 


Within  your  group,  discuss  the  types  of  "x"  factors  that  your  projects 
within  small  settings  would  be  most  affected  by.  These  should  be 
factors  related  to  the  people,  process,  tools,  technology  or  environment 
that  most  affect  or  determine  the  performance  outcomes. 


Document  the  ideas  on  your  group  flip  pad.  Be  sure  to  distinguish  the 
controllable  vs  un-controllable  "x"  factors. 


Be  prepared  to  share  some  of  these  factors  with  the  audience  at  large 
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Congreso  Internacional  en  Ingenieria  de  Software  y  sus  Aplicaciones 
(International  Congress  of  Software  Engineering  and  its  Applications) 


CONTRASTING  LARGE  VS 
SMALL  ORGANIZATIONAL 
SETTINGS: 

USAGE  OF  MODELS 
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Usage  of  Models 


Large  Settings 

•  Statistical  management  of 
key  subprocesses  usually 
related  to  key  handoffs  in 
large  teams 

•  Predict  outcomes  at  key 
milestones  or  end  of  key 
phases 

•  Support  significant  CAR  or 
OID  activity 


Small  Settings 

•  Provide  updates  on 
impacts  of  key  technology 
or  people  issues 

•  Predict  updated  impacts  on 
key  risks  based  on  real¬ 
time  information  or  events 

•  Predict  "what-if's  for  real¬ 
time  replanning  during 
weekly  if  not  daily  intervals 

Predict  abilities  on  a 
feature  by  feature  basis 
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Group  Exercise  #4 
(10  minutes) 


Congreso  Internacional  en  Ingenieria  de  Software  y  sus  Aplicaciones 
(International  Congress  of  Software  Engineering  and  its  Applications) 


Within  your  group,  discuss  the  usage  of  process  performance 
models  that  your  projects  within  small  settings  would  most  likely  use. 
Be  sure  to  note  the  rationale  for  the  analytical  models  identified. 


Document  the  model  ideas  on  your  group  flip  pad 
Be  prepared  to  share  some  of  these  with  the  audience  at  large 
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Congreso  Internacional  en  Ingenieria  de  Software  y  sus  Aplicaciones 
(International  Congress  of  Software  Engineering  and  its  Applications) 


CONTRASTING  LARGE  VS 
SMALL  ORGANIZATIONAL 
SETTINGS: 

ANALYTICAL  METHODS 
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What  Is  a  Statistic? 


Congreso  Internacional  en  Ingenieria  de  Software  y  sus  Aplicaciones 
(International  Congress  of  Software  Engineering  and  its  Applications) 


A  summary  or  characterization  of  a  distribution  (i.e.,  a  set  of  numbers) 


A  characterization  of  a  central  tendency  (e.g.,  mean,  median,  and 
mode) 

A  characterization  of  Useful  Probabilities  for  Normal  Dis 

_  6  8  %  _ 

dispersion  (e.g.,  variance,  j  1  95o/° _ j _ 

standard  deviation,  j - 1 - f —  99%  — - 

interquartile  range,  and  range) 


Congreso  Internacional  en  Ingenieria  de  Software  y  sus  Aplicaciones 
(International  Congress  of  Software  Engineering  and  its  Applications) 


Central  Tendency  and  Dispersion 


Central  tendency  implies  location: 

•  middle  of  a  group  of  values 

•  balance  point 

•  examples  include  mean,  median,  and  mode 
Dispersion  implies  spread: 

•  distance  between  values 

•  how  much  the  values  tend  to  differ  from  one  another 

•  examples  include  range  and  (sample)  standard  deviation 

These  two  are  used  together  to  understand  the  baseline  of  a  process- 
performance  factor  and  outcome. 
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Congreso  Internacional  en  Ingenieria  de  Software  y  sus  Aplicaciones 
(International  Congress  of  Software  Engineering  and  its  Applications) 


Hypothesis  Testing:  To  Understand  and 
Compare  Performance 


A  formal  way  of  making  a  comparison  and  deciding  whether  or  not  the 
difference  is  significant  is  based  on  statistical  analysis. 

Hypothesis  testing  consists  of  a  null  and  alternative  hypothesis: 

•  The  states  that  the  members  of  the  comparison 

are  equal;  (a  concrete,  default  position). 


•  The  alternative  hypothesis  states  that  there  is  a  difference;  it  is 

supported  when  the  null  hypothesis  is  rejected. 


The  conclusion  either  rejects  or  fails  to  reject  the  null  hypothesis. 


Understanding  the  null  and  alternative  hypotheses  is  the 
key  to  understanding  the  results  of  statistical  prediction 
models. 
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Congreso  Internacional  en  Ingenieria  de  Software  y  sus  Aplicaciones 
(International  Congress  of  Software  Engineering  and  its  Applications) 


Formally  Stating  a  Hypothesis 


Average  productivity  equals  100  source  lines  of  code  (SLOC)  per  person 
week: 


•  Null:  Average  productivity  is  equal  to  100  SLOC  per  person  week. 

•  Alternative:  Average  productivity  is  not  equal  to  100  SLOC  per 
person  week. 

A  refinement  of  these  hypotheses  are  as  follows: 

•  Null:  Average  productivity  is  equal  to  100  SLOC  per  person  week. 

•  Alternative:  Average  productivity  is  less  than  100  SLOC  per  person 
week. 

Generally,  the  alternative  hypothesis  is  the  difference  (e.g.  improvement  or 
performance  problem)  that  you  seek  to  learn  about. 


The  null  hypothesis  holds  the  conservative  position  that  apparent 
differences  can  be  explained  by  chance  alone.  The  phrase  “is  equal  to” 
will  generally  appear  in  the  null  hypothesis. 
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We  Must  Understand  Distributions  -  They  are  Key 
to  Informed  Decisions 
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Congreso  Internacional  en  Ingenieria  de  Software  y  sus  Aplicaciones 
(International  Congress  of  Software  Engineering  and  its  Applications) 


Distributions  Describe  Variation  in  Process  Factors 

Populations  of  data  may  be  viewed  as  distributions  in  statistical  procedures: 

•  expressed  as  an  assumption  for  the  procedure 

•  can  be  represented  using  an  equation 

The  following  are  examples  of  distributions  you  may  come  across: 


[£ 

i 

EJ 

Normal 


Triangular 


Uniform 


Lognormal 


Beta 


Gamma 


Weibull 


C  □  [C 

Max  Extreme  Min  Extreme  Logistic 


Student's  t 


Exponential 


L_ 

Pareto 


j 

III,.. 

..ill 

III. 

,1 

ll.. 

ll  . _ 

Poisson 

Hypergeometric 

Neg  Binomial 

Geometric 

Discrete  Uniform 


tl 

Yes-No 


Custom 
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Congreso  Internacional  en  Ingenieria  de  Software  y  sus  Aplicaciones 
(International  Congress  of  Software  Engineering  and  its  Applications) 


Monte  Carlo  Simulation  Models  Process  Factors 


We  can  identify  process  factors  that  have  uncertain  distributions  of 
behavior 


Then  we  can  load  them  in  a  spreadsheet  and  calculate  the  predicted 
performance  outcomes 


The  performance  outcomes  will  also  have  distributions  of  behavior 
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A 


2  3 


1  2  3  4  5 

Crystal  Ball  then 
allows  the  user  to 
analyze  and  interpret 
the  final  distribution 
of  C! 


Congreso  Internacional  en  Ingenierla  de  Software  y  sus  Aplicaciones 
onal  Congress  of  Software  Engineering  and  its  Applications) 


~^n; 


Crystal  Ball  uses  a 
random  number 
generator  to  select 
values  for  A  and  B 


A  +  B 


123456789  10 
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B 


2  3 


1  I  2  3 


1  2  3  4  5 


Crystal  Ball  causes 
Excel  to  recalculate 
all  cells,  and  then  it 
saves  off  the 
different  results  for 
C! 
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Developing  Correlation  and  Regression  Models 


Continuous 

/ - 


Discrete 


0) 

0> 

s— 

O 

c n 

Q 


c n 

13 

O 


AN  OVA 
&  Dummy 
Variable 
Regression 

Chi-Square 
&  Logistic 
Regression 

Correlation 
&  Simple 
Regression 

Logistic 

Regression 
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Analytical  Methods 


Large  Settings 

•  Large  investment  in 
discrete  event  process 
simulation  models  for 
complex  processes 

•  Large  collection  of  process 
performance  models  to 
deal  with  most  phases  and 
key  activities/hand-offs 


Small  Settings 

•  Small  regression  equations 

•  Small  probabilistic  models 

•  A  greater  use  of  Monte 
Carlo  simulation  for  real¬ 
time  assessment  of 
unbalanced  risk 

•  A  small  number  of  process 
performance  models 

•  Models  built  and  operated 
within  individuals 
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Group  Exercise  #5 
(10  minutes) 


Congreso  Internacional  en  Ingenieria  de  Software  y  sus  Aplicaciones 
(International  Congress  of  Software  Engineering  and  its  Applications) 


Within  your  group,  discuss  the  types  of  analytical  methods  that  your 
projects  within  small  settings  would  most  likely  use. 


Document  the  types  of  analytical  methods  on  your  group  flip  pad 


Be  prepared  to  share  some  of  these  with  the  audience  at  large 
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Congreso  Internacional  en  Ingenieria  de  Software  y  sus  Aplicaciones 
(International  Congress  of  Software  Engineering  and  its  Applications) 


CONTRASTING  LARGE  VS 
SMALL  ORGANIZATIONAL 
SETTINGS: 

TRAINING  AND  DEPLOYMENT 
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Training  and  Deployment 


Large  Settings 

•  Corporate  deployment 
team 

•  Develop  training  internally 
or  purchase  expensive 
external  training  materials 

•  Hire  a  team  of  experienced 
deployment  change 
agents 

•  Send  waves  of  people  thru 
external  training 


-  Software  Engineering  Institute 


Small  Settings 

•  Identify  a  few  experts  to 
receive  training 

•  Identify  a  few  consultants 
or  external  coaches  to  help 
when  needed 

•  Hitch  a  ride  on  training 
and/or  consulting  that  a 
larger  organization  is 
conducting  (commercial  or 
gov't  agency) 


Carnegie  Mellon 
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Group  Exercise  #6 
(10  minutes) 


Congreso  Internacional  en  Ingenieria  de  Software  y  sus  Aplicaciones 
(International  Congress  of  Software  Engineering  and  its  Applications) 


Within  your  group,  discuss  the  training  and  deployment  that  your 
projects  within  small  settings  would  most  likely  pursue.  Identify  the 
aspects  of  training  and  deployment  that  your  projects  would  most  likely 
be  concerned  with. 


Document  the  ideas  on  your  group  flip  pad 


Be  prepared  to  share  some  of  these  with  the  audience  at  large 
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Congreso  Internacional  en  Ingenieria  de  Software  y  sus  Aplicaciones 
(International  Congress  of  Software  Engineering  and  its  Applications) 


CONTRASTING  LARGE  VS 
SMALL  ORGANIZATIONAL 
SETTINGS: 

SPONSORSHIP  AND 
PARTICIPATION 


=  Software  Engineering  Institute  Carnegie  Mellon 
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Congreso  Internacional  en  Ingenieria  de  Software  y  sus  Aplicaciones 
(International  Congress  of  Software  Engineering  and  its  Applications) 


Traditional  Management  Review  Perspective 


Management  has  come  to  realize  that  just  looking  at  the  customary  lagging 
outcomes  is  like  driving  a  car  using  only  the  rear-view  mirror. 


Excerpted  from  the  SEI  course  called 
“Understanding  CMMI  High  Maturity  Practices” 
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Congreso  Internacional  en  Ingenieria  de  Software  y  sus  Aplicaciones 
(International  Congress  of  Software  Engineering  and  its  Applications) 


High  Maturity  Management  Review  Perspective 


Management  dashboards  in  High  Maturity  organizations  include  not  only 
outcomes  but  leading  indicators 


-  such  as  the  controllable  x  factors 
used  in  process  performance 
models. 


Thus,  management  has  asked  for 
an  additional  3-5  leading  indicators 
for  each  traditional,  lagging 
indicator  used  on  dashboards. 


Excerpted  from  the  SEI  course  called 
“Understanding  CMMI  High  Maturity  Practices” 
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Congreso  Internacional  en  Ingenieria  de  Software  y  sus  Aplicaciones 
(International  Congress  of  Software  Engineering  and  its  Applications) 


A  Change  in  Senior  Management  Behavior 


Before,  management  spent  approx.  80%  of  each  management  review 
looking  at  the  lagging  indictors  (e.g.  the  outcomes  of  cost,  schedule 
and  quality) 


Now,  in  High  Maturity,  they  spend  approx.  80%  of  their  time  reviewing 
the  statistical  management  of  controllable  x  factors  and  the  results  of 
process  performance  model  predictions. 


The  discussion  is  now  primarily  focused  on  how  management  can  pro¬ 
actively  take  action  based  on  performance  models  predictions! 


Excerpted  from  the  SEI  course  called 
“Understanding  CMMI  High  Maturity  Practices” 


Interpreting  CMMI  High  Maturity  for  Small 
Organizations 
Robert  W.  Stoddard  II 
September,  2008 

©2008  Carnegie  Mellon  University 


Software  Engineering  Institute  Carnegie  Mellon 


58 


A  Change  in  Management  Review  Charts 
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Excerpted  from  the  SEI  course  called 
“Understanding  CMMI  High  Maturity  Practices” 


Success  Indicators 

(Lagging  Indicators) 


Progress  indicators 

(Lagging  Indicators) 


For  project 
manager 


Reporting  Periods 


Roll-up  for 
higher  management 


Actual 

X 


Reporting  Periods 
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Sponsorship  and  Participation 


Large  Settings 

•  Significant  top-down  sales 
pitch  to  executives  and 
middle  management  is 
required 

•  Dedicated  resources 
provide  full-time  support  for 
key  modeling  activities 

•  Key  process  owners  get 
involved  but  not  average 
developer 


Small  Settings 

•  Top-down  or  bottom-up 
approaches  can  work 

•  Success  will  breed 
success  (show  early 
benefit) 

•  Most  individuals  will  be 
involved  with  the  basic 
modeling  techniques 

•  A  single  person  may  serve 
as  a  coach  for  rest  of  team 


-  Software  Engineering  Institute  Carnegie  Mellon 
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Group  Exercise  #7 
(10  minutes) 


Congreso  Internacional  en  Ingenieria  de  Software  y  sus  Aplicaciones 
(International  Congress  of  Software  Engineering  and  its  Applications) 


Within  your  group,  discuss:  1 )  the  challenges  with  management 
sponsorship  and,  2)  the  degree  of  team  participation  that  your 
projects  within  small  settings  would  most  likely  experience. 


Document  the  ideas  on  your  group  flip  pad  along  with  ideas  on  how 
you  would  prevent  or  mitigate  these  issues. 


Be  prepared  to  share  some  of  these  with  the  audience  at  large 
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Congreso  Internacional  en  Ingenieria  de  Software  y  sus  Aplicaciones 
(International  Congress  of  Software  Engineering  and  its  Applications) 


NEXT  STEPS 
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Ideas  for  Next  Steps 


Congreso  Internacional  en  Ingenieria  de  Software  y  sus  Aplicaciones 
(International  Congress  of  Software  Engineering  and  its  Applications) 


Identify  how  to  integrate  a  CMMI  High  Maturity  approach  with  existing 
improvement  methods  (specifically  TSP/PSP  provide  a  strong 
measurement  culture  to  support  process  performance  modeling) 

Identify  and  acquire  the  necessary  training  and/or  skilled  staff  for 
CMMI  process  performance  modeling  (consider  an  integration  of 
certified  CMMI-Six  Sigma  Belts  in  addition  to  certified  PSP  Developers 
and  TSP  coaches) 

Hold  necessary  workshops  to  identify  compelling  business  and  project 
level  performance  and  guality  goals  (SEMA  offers  a  jumpstart 
workshop  on  this) 

Develop  process  performance  models  and  institutionalize  their  usage 
and  maintenance  (SEMA  offers  hands-on  coaching  of  this) 
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Congreso  Internacional  en  Ingenieria  de  Software  y  sus  Aplicaciones 
(International  Congress  of  Software  Engineering  and  its  Applications) 


SEI  Measurement  Curriculum 


Course  Title 

•Implementing  Goal-Driven  Measurement 

•Yellow  Belt 

•Analyzing  Project  Management  Indicators 

•Improving  Process  Performance  Using  Six  Sigma 

•Green  Belt 

•Designing  Products  and  Processes  Using  Six 

•Black  Belt 

Sigma 

•Living  the  High  Life:  A  CMMI  High  Maturity  Tutorial 

•Understanding  CMMI  High  Maturity  Practices 

•Black  Belt 
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Congreso  Internacional  en  Ingenieria  de  Software  y  sus  Aplicaciones 
(International  Congress  of  Software  Engineering  and  its  Applications) 


SEI  CMMI-Six  Sigma  Belt  Certification  Program 


Certifications  — 


Certificate 


CMMI-Six  Sigma  Master  Black  Belt 


CMMI-Six  Sigma  Black  Belt 


CMMI-Six  Sigma  Green  Belt 


CMMI-Six  Sigma  Yellow  Belt 
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Congreso  Internacional  en  Ingenieria  de  Software  y  sus  Aplicaciones 
(International  Congress  of  Software  Engineering  and  its  Applications) 


Preliminary  Qualification  Requirements 


Education,  Experience, 
Competency,  and  Skills 
Clusters 

SEI  Designation  in 
Yellow  Belt 

SEI-Certified  CMMI- 

Green  Belt 

SEI-Certified  CMMI-Six  Sigma 
Black  Belt  in  CMMI 

SEI-Certified  CMMI-Six 

Master  Black  Belt 

Prerequisite  to  enter 
qualification  track 

None 

SEI  CMMI-Six 
Sigma  Yellow  Belt 

SEI  CMMI-Six  Sigma  Green 
Belt 

SEI  CMMI-Six  Sigma  Black 
Belt 

CMMI 

Introduction  to 

CMMI  v  1.2 

Intermediate 

CMMI : 

or  pass  the 
Intermediate 
Concepts  for 

CMMI 

examination 

Understanding  CMMI  High 
Maturity  Concepts: 
or  become  certified  HM 
Lead  Appraiser 

Participate  as  an 

Appraisal  Team  Member 
on  Two  (2)  SCAMPI  A  or  B 
appraisals 

CMMI-Six  Sigma 

Strategies 

Measurement  and 

Analysis  &  Six  Sigma 

Implementing 

Goal  Driven 

Measurement 
(IGDM)or 
complete  the 

IGDM  Exercise 

Improving  Process 
Performance 

Using  Six  Sigma 
(IPPSS) 

Designing  Process  and 
Products  using  Six  Sigma 
(DPPSS) 

-Attend  1  Phase 

Transition  Workshop 
-Lead  a  min  of  1  Phase 

Transition  Workshop 
mentored  by  a  SEI 

Certified  MBB 

Electives: 

(Present  evidence  of 
completion) 

Complete  one 
course  related  to 
statistically  based 
problem  solving 
approaches 

Show  evidence  of 
successful  completion  of 
one  of  the  following  SEI 

courses: 

-SEI  Mastering  Process 
Improvement 

-  SEI  Managing 
Technological  Change 

-  CMMI-Six  Sigma 

Strategies 

-Show  evidence  of 
Mentoring/Coaching 

Teams  Training 
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